How to Create An Optimization Process? A Guide to Building a Culture of Experimentation That Supports Consistent Growth

A test by test optimization program is not going to cut it if you want to achieve continuous growth. In 2019, CXL reported that 38.3% of optimizers have an undocumented or unstructured process while 17.1% have no process whatsoever. Proper optimization that drives sustainable growth is a process that requires a well-defined structure to run and maintain.

Building a conversion rate optimization program that allows your team to constantly test and improve will not be a walk in the park. Building an optimization process is one of the things many optimizers suffer from. There are many challenges to overcome.

First, there is the mindset that sees optimization as a marketing activity to be run only when conversions need a boost or dip below acceptable standards. While recognising the benefits of optimization, there isn’t a consistent experimentation framework in your company.

Secondly, there is the age of your optimization program. If CRO practice is still young in your company, it can be daunting establishing a framework whilst running tests and trying to get wins. There is the temptation to forgo creating an optimization process now that you are just getting started. But this will come back to haunt you as your optimization program matures.

Optimization is a young industry. The 2019 CXL study states that 60% of optimizers have been working in the industry for only 2 years. An optimization framework will help your company get the most out of the optimizers in your team and push you towards an optimization program that drives continuous growth.

There are many more challenges to surmount. So, let’s show you how to build an optimization structure that supports continuous growth!

Adopt an Experimentation and Optimization Mindset

Steve Maraboli said:

Once your mindset changes, everything on the outside will change along with it.

To build a conversion optimization structure that works for your company, you need to nurture the right mindset. A general mindset geared towards optimization creates an excellent foundation you can lay your conversion rate optimization framework on.

Here are some ethos that will make your company more focused on optimization.

Be Data Driven

Since split and multivariate testing are driven by quantitative (analytic) and qualitative data, a shift to a data first mindset will do your company a world of good.

A data-driven mindset helps your company not only optimize consistently, but also influences other business decisions like diversifying into a new market or predicting future trends.

The great thing about being data driven is that your marketing team already operates that way. If your marketing team uses analytic data to inform actions such as updating content to capture more traffic, create buyer personas, then they are already using a data-first approach. Other teams in your company are probably using data to make their workflow and processes easier. If not, then get them on the same page.

Take an Agile Approach

Often, the expectation in many companies is that results will have a huge impact. Employees are under tremendous pressure to deliver this perfect unicorn. This often leads to delays as perfection takes precedence over done.

This attitude is at odds with optimization. In optimization, a small win that you can get within a week is better than waiting a year to get a larger win. A/B testing a landing page today and getting results in weeks is better than waiting 3 months to conceive, plan and create the perfect multivariate test that solves nearly every problem on your website in one swoop (is that even possible?). Not that complex tests are bad. The idea is to start with small tests and work your way up to more complex tests.

A shift from the “go big or go home” working mindset is hard to achieve. But it can be done. Start by emphasizing progress and finishing tasks (doesn’t have to be perfect) in your team. This will force a pivot from chasing perfection to getting things done, no matter how small. Like we say at Convert, Progress not Perfection.

Learn From Failure

Winning is exhilarating and addictive. Once you get your first win, you want to stay on a streak of wins. Failing is no longer an option you are comfortable with.

But optimization doesn’t see failure as a bad thing. When you run tests, some of your control will beat the variations. While you may see this as a failed test, it really isn’t. These “testing failures” provide unique business insights into your customers’ behavior. It also challenges your preconceived notions and helps you come up with better hypotheses.

Normalize failures as features of making progress and learn from them. This will shift your company’s mindset into optimization first.

Open up experimentation to your entire company

Your experimentation program should not just be limited to your CRO team. An experimentation mindset is one you want to cultivate in your entire company. And opening up the process for other teams in your company to run experiments will vastly improve your optimization program.

Take Booking.com for example, they have a robust experimentation program that allows individuals from different teams in the company to run tests by filling standardized templates. Name of experiment, results, learnings and iterations are stored and easily searchable in a database. 

Opening up experimentation can have a few challenges like an individual’s test may break something on the site. But having your core experimentation team keep an eye on what is going on will prevent this. 

Opening up experimentation to your whole company helps build an experimentation culture and mindset that other teams can bring into their areas of expertise

Take Risks

Risk is an inherent part of life and business. To be successful in any endeavor, take some risks.

Does your company take risks?

Some companies are very risk averse. They achieve early success and become reluctant to take chances. Others may just not enjoy taking risks since there is always a chance that the company may lose.

A proper optimization program involves taking risks. Experimentation often involves taking risks and testing radical changes that may succeed or fail. If the variation wins, it is implemented and everyone celebrates the win. If the control beats the variations, the hypothesis is discarded and the lesson from that test is used to inform subsequent tests.

A risk conservative attitude as an optimization-forward company usually equals testing small safe changes even in the face of contradicting data. This reluctance to test radical changes often means your business loses out on larger lifts. And it could also mean losing market share to competitors who are ready to take a chance on testing major changes.

One easy way to improve your risk appetite is to stop being comfortable with where your optimization is and discard the fear of failure. Start with small risks and work your way up. Greater risks often equal greater rewards.

Set the Right KPIs for Experimentation

The purpose of your optimization program is to drive growth and achieve other business goals. 

What exactly does growth look like in your company?

Growth may mean more revenue, purchases, form sign-ups, subscriptions, comments, or pageviews. Every company operates in different niches and thus, growth will be different for each one. The first step is to define what growth means to your business. Then attach a metric that best measures what your company defines as growth. Say you run an ecommerce business and you define growth as increase in revenue, average revenue per visitor will be a good metric to attach to your definition of growth which is more revenue. 
With a way to measure business growth in hand, let’s dive further into primary and secondary KPIs for your experiments.

Choosing Primary and Secondary KPIs

Different experiments have different hypotheses they test. One experiment may aim to test changes to a page to reduce bounce rate, another may test form fields to increase form submission, another may test changes that hopes to increase revenue per user, etc. Because each experiment is testing different things, you need to choose a primary KPI. 

A primary KPI is a primary metric for which you plan an experiment for and judge its outcome by. An experiment to reduce bounce rates may use bounce rate as a primary KPI while another to increase revenue may have average revenue per user. In a bounce rate experiment, a reduced bounce rate will mean your hypothesis is right and you can deem your experiment a rousing success. 

While it is not the main metric you designed your experiment for, a secondary KPI may be something you are interested in measuring. In an experiment to increase average revenue per visitor, you may be interested in measuring time spent on page as well. While average revenue per user is your primary KPI, time on page can be your secondary KPI. Secondary KPIs do not affect when you stop and start experiments nor do they affect how you judge the outcome of an experiment. 

When choosing both primary and secondary KPIs, make sure that both KPIs serve the main goals of your organization. If your goal is more revenue, your KPIs should be a reflection of that.

Define Your Optimization and Experimentation Process

Now that your entire team and business has adopted an optimization first mindset, it is time to create an optimization process for experimentation program.

When creating your experimentation process, you will be inundated with images like:

You may be tempted to base your framework on such images, but these frameworks were not created with your company in mind. Most online experimentation process diagrams do not take into account unique factors like your website maintenance schedule, buyer journey, how your marketing is set up and seasonality of your traffic/business.

These are important factors that affect your optimization efforts. An online optimization process will not cut it.

Here is how to create and define an experimentation process that is tailored to your business:

1. Define the Problem

The first step in your experimentation program is to define the problem. You absolutely can’t optimize a “perfect website”. For optimization to happen, there has to be something to fix. You may be trying to increase revenue, decrease churn, provide better experience so you can upsell current customers etc. There is always room for improvement.

Data already informs many processes in your marketing workflow. Analytic data is probably telling you that a landing page, for example, has a low conversion rate when compared with other pages on your website. This data points out a problem which you can easily define as “low conversions”.

Knowing and defining the problem means you can make it better.

You absolutely can’t optimize a “perfect website”. For optimization to happen, there has to be something to fix. You may be trying to increase revenue, decrease churn, provide better experience so you can upsell current customers, etc. There is always room for improvement.

Data already informs many processes in your marketing workflow. This data can come from analytics software, heatmap tools, customer feedback etc. This data points to a problem or an area that needs to be improved. For example, your analytics software may have pointed to low conversion rates on a high traffic landing page. 

Once you have identified an area that needs improvement, it is time to research the problem and find a solution.

Knowing and defining the problem means you can make it better.

2. Research the Problem

When your data says there is a problem or something to fix, how do you get more insights about the problem?

By collecting more data about the problem, of course!

In our previous example, the data showed low conversion rates on a landing page that is getting a lot of traffic. You need to find out what exactly makes visitors not convert on that page. 

This data can come from a mixture of both quantitative and qualitative sources. You can use form analysis, session recording, usability research, customer feedback, customer interviews, etc. to investigate the problem further. 

With more insights, you can understand why your high traffic landing page has a low conversion rate. 

After using session recording and polls on your low conversion rate landing page, you find that visitors find the visual elements on the page too distracting. Now what?

3. Form a Hypothesis

A hypothesis is an educated guess about what will fix the problem you have identified on your website. Fixing this problem means more visitors will take the desired action and thus, improve the conversions on that page/site.

In the previous example, you have identified the problem as distracting visual elements. You form a hypothesis that removing visual elements will increase the conversions on the landing page.

In the real world, there is never just one page that needs improvements. Often, it is multiple pages and sites that need improvements.

This is where prioritization comes in.

With hypotheses on multiple pages, decide which hypotheses are important and need to be tested first. You can do this using either the PIE or ICE prioritization models.

In the PIE prioritization model, you score your hypothesis on 3 different factors using a score of 1-5 with 5 being the highest score. These factors are:

• Potential for improvement: how likely the hypothesis will lead to an improvement on the test page.
• Importance: the value of the traffic landing on the test page
• Ease: the rate of difficulty of implementing the changes from the hypothesis.

The ICE prioritization model uses a similar system. But its factors are:

• Impact: a measure of the positive impact of your hypothesis on the metric you want to improve.
• Confidence: how sure you are about the impact of this hypothesis.
• Ease: an estimate of how many resources will be needed to implement the changes the hypothesis calls for.

Keeping track of multiple hypotheses and their various prioritization scores can be tough. You can use Convert Compass to stay on top of things. It enables you to create hypotheses and assign prioritization scores to them based on either model.

Give your hypotheses status based on the stage of their conception: draft, completed, applied to an experiment and archived. You can see and manage all your hypotheses in one dashboard.

4. Design Variations and Conduct an Experiment

After creating your hypotheses and giving them prioritization scores, you can design and conduct experiments on the hypotheses with high scores. Usually, this experiment will take the form of A/B testing where you have your control, like the landing page in the example, and a variant (with less visual elements) which you designed based on a hypothesis.

With your site’s seasonality factored in, you can now launch your experiment. In Compass, you can turn any of your completed hypotheses to an Experience within seconds.

This makes it easy to conduct tests on your hypotheses all within the Convert Experiences dashboard. Set the number of days your experiment will run andstatistical significance by clicking on Stats & Setting in Experiment Summary and editing the options.

5. Analyze the Results and Reach a Conclusion

At the end of the experiment comes an analysis of the results. Are your results statistically valid? In a 2019 analysis of over 28,000 experiments, we found that only 20% reached the 95% statistical significance level. There was a tradeoff between statistical significance and experiment speed.

You can set your confidence level as outlined above in Convert Experiences. Now you can delve into the data your experiment has gathered.

Did your variation perform against the control? Was your hypothesis right? The results will answer these questions and more.

Robust reporting makes analyzing the results easier. The ability to tell from a glance which test won in the experiment, its statistical validity, goals and metrics, etc.

After analyzing the results, it should be clear what the conclusions are. If your variation beats your control, the conclusion is that your hypothesis was right and thus, the changes will be implemented. If the control won, the conclusion is that the hypothesis was wrong and may need modification. The insights from the experiment can fuel subsequent experiments.

6. Socialize your Results

This is an important piece of your optimization program Since you’re building an experiementation process that will support continuous improvements, socializing your results is vital. It will help increase tribal knowledge of optimization in different teams in your company. The teams in your company (product, dev, customer and sales etc.) should be able to understand what optimization, why it’s important, how it works and how it helps build sustainable growth. Socializing your results helps you achieve this.

Apart from sharing your results, you can even take socialization further by throwing defining problems and hypotheses to other departments. Have other teams in your company provide hypotheses for testing. These experiments can be labeled by team name and results socialized after the experiment has ended to deepen your company’s appreciation of experimentation and promote an optimization mindset.

Handy graphs that can be exported will make socialization easier as teams can see the performance of their variations at once.

7. Here Comes a Caveat

This framework is a flexible guide you can adapt to your organization. Remember, there will be situations that are so high priority that you may not need to design variations and test because you are losing valuable traffic, revenue, etc. In those situations, you go from hypothesis and prioritization straight to implementing a change and pushing it live. 

For example, you have a high-traffic page with a broken link that is necessary for visitors to convert. After seeing no conversions on this page and investigating the cause, you form a hypothesis and assign it a high prioritization score. 

What do you do in this case?

1. Will you rigidly follow the optimization framework by designing a variation and testing it?
2. Will you fix the broken link on your page and push it live?

The answer is obviously the second option. Designing a variation and testing it will waste resources and cost you conversions. 

Take the time to consider each problem or improvement to your website that crops up. Every hypothesis will not become an experiment. Consider the opportunity cost and importance of every hypothesis before turning it into an experiment. 

You can still socialize the results of making a change that skipped several steps in the optimization process. Just remember that this framework is flexible enough to follow different paths.

Skillsets Your Optimization Team Should Have

With a powerful CRO experimentation process in hand, perform a skills analysis of your optimization team. Your team is just as important as having an experimentation framework nailed down.

Here are skills vital to an optimization team:

• Marketing acumen
• Data analysis
• Heuristic/Usability/UX design
• Visual design
• Optimization expert
• Copywriting
• Front-end development
• Project Management

Each skill plays a different role in your optimization program. Visual design, for example, creates variations that look different and conveys information in different ways. An optimization expert understands testing, can create an optimization strategy, the technology involved, methodology, statistics and more. Front-end development brings knowledge of Javascript, JQuery, how tags work and how browser rendering works.

These sounds like a lot to make a team but these skills are already present in your team. An analysis of the skills of your marketing team will assure you of this. Your marketing team most likely has a copywriter, someone who lives and breathes Google Analytics, a designer, a developer, etc.

Even if you don’t have all the skills needed in an optimization team, you don’t have to jump straight to hiring a new team member.

First, identify the skills lacking and ask yourself:

• Can I train someone in-house to fill this skill gap?
• Will they have enough time to take up the additional responsibility? If not, should I increase their hours?
• Can I outsource these skills? How do I identify trusted experts to fill these roles? At Convert, we have a vetted directory of great optimization partners that you can outsource to without any worry.
• Will I eventually need to hire someone?

These questions will help you narrow down a course of action. At Convert, we recommend having an optimization expert, a developer and a personalization expert if you run personalization experiments.

CRO Tools Needed in Your Optimization Structure

Good news, your optimization structure is nearly complete. No framework is complete without tools to power it. The tools you pick can either enable your optimization structure or make the work of your team a lot harder.

You need a tool to:

• Coordinate resources
• Design variations
• Document requirements
• Conduct experiments
• Analyze the results data
• Socialize experiments results

These look like you need a lot of new tools in your marketing stack. But you don’t. Chances are you manage tasks using an Asana or Basecamp or Trello. Your testing tool should be able to handle most if not all the rest.

In Convert Experiences, you can document your hypotheses, turn them into experiments and analyze the results. You can export the data in Convert Experiences to any analytics software of your choice for a deep dive into the results data. Convert integrates with Google Analytics (both Classic & Universal), Heap Analytics, Amplitude Analytics, Adobe Analytics, Decibel Insight and a lot more. Take a dive into Convert Experiences 80+ Integrations. 

You can also share the results of your experiment with other members of your company via the dashboard to socialize your results and deepen tribal knowledge.

Wrapping up

Building a structure that supports consistent optimization in your company requires intentional action, an excellent experimentation framework, a great tool stack and impressive skills.

The important thing is to take the steps to build your optimization program, no matter how small. Remember to practice the mindset shift you have adopted, nurture the skills in your team and use an awesome CRO tool for your testing solution.

Take the first step in building a strong optimization framework by trying Convert Compass and Convert Experiences for free for 15 days .(No pesky credit cards required!)

Convert Compass + Experiences in Trial